Three dimensional (3d) graphics, in particular, simulated 3d realms or worlds, sometimes called 3d virtual reality, is a well known area of computer graphics, which typically involves rendering two dimensional images of 3d models and scenery in a 3d coordinate space. Most modern game consoles are designed specifically to be able to process 3d graphics in real-time, and many games for game consoles are based on a simulated 3d or virtual reality.
Game consoles are usually operated using game controllers, such as joysticks, button pads, and so on. For many players, significant time must be spent before a game control can be used proficiently. For lack of dexterity, many people do not regularly use a game console. Although game consoles have become powerful and can process a high level of control input from a user, it is difficult to provide a high level of control input using typical game controllers such as a touchpads, joysticks, mice, etc. Furthermore, game controllers are often not physically operated in a manner that meets a user's mental model or intuitive understanding of a game. In other words, a 3d game may involve causing an avatar to run, jump, hold different objects, shoot, duck, etc., perhaps simultaneously. However, a game controller can usually only output one or a few input signals at one time, and hands are not efficient at simultaneously controlling different types of actions, such as simultaneously moving an avatar, picking up an object, aiming, and so on.
Game consoles have also been limited in the type of imagery that they display. Typically, the graphics displayed by a game console are generated internally by a game or are provided in advance. For example, the images for animating a sprite may be part of a game's embedded content or program data. Real-time captured video of objects has not been inserted or mapped into 3d virtual reality.